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ABSTRACT 

This paper describes a f our~dimensional model for the 
science of knowledge acquisition and a classification system in terms 
of these dimensions. A summary of the state of the art in terms of 
the cells generated in a four-dimensional representation of this 
classification system is presented. The need for better tools for 
conducting knowledge classification efforts in many of these cells is 
shown. One such tool, derived from a teaching model called Structural 
Communication, is described. It is argued that this tool would be 
useful in many of the cells for which no tools currently are 
available. An experiment using the Structural Communication approach 
to knowledge acquisition at the inter-cultural le^'el is described. 
The resulting knowledge (i.e^, data) is anlayzed using a method 
called Pattern Noting. The results from the data analysis are 
discussed in terms of what they show concerning the usefulness of the 
approach to knowledge acquisition, and recommendations for further 
research and application are made. (Contains 47 references.) (ALF) 
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ABSTRACT 



This paper describes a four -dimensional model for the 
science of knowledge acquisition, and a classification system in 
terms of these dimensions. A summary of the state of the art in 
terms of the cells generated in a four-dimensional representation 
of this classification system is presented. The need for better 
tools for conducting knowledge classification efforts in many of 
these cells is shown. One such tool, derived from a teaching 
model called Structural Communication, is described. It is 
argued that this tool would be useful in many of the cells for 
which no tools currently are available. An experiment using the 
Structural Communication approach to knowledge acquisition at the 
inter-cultural is described. The resulting knowledge (data) is 
analyzed using a method called Pattern Noting. The results from 
the data analysis are discussed in terms of what they show 
concerning the usefulness of the approach to knowledge 
acquisition, and recommendations for further research and 
application are made. 



THE KNOWLEDGg ACQUIS I TION PROBLEM 



"Knowledge acquisition" is the acquisition of a domain of 
knowledge from its current location and its restructuring so as 
to make possible the insertion of this knowledge into a second 
location. Knowledge acquisition is not the only operation that 
can be performed on a domain of knowledge; the second operation 
is "knowledge creation", "knowledge production", or "Knowledge 
generation". "Knowledge generation" is the production of domain- 
specific content that was not previously present in that domain. 
A third type of operation on knowledge is often called "knowledge 
transmission". This term refers to the link between the knowledge 
producer and the knowledge user; in schools, this is called the 
"teaching/ learning process"; in the larger cultural context, the 
terms "diffusion" and "adoption" are used. All of these terms 
refer to basically the same process, which is \indertaken to deal 
with the same problem. 
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The problem is how to acquire a conceptual structure that 
was not previously available. At the applied, technical level. 
Knowledge acquisition is concurrently both the most important and 
the most problematic aspect of expert system development (McGraw 
and Harbison-Briggs> 1989). Knowledge acquisition efforts have 
failed to yield themselves to automation, or even to straight- 
forward manual methodologies. It is the area of knowledge 
acquisition that forms the bottleneck within most large expert 
systems development projects. And not only a large proportion of 
development time, but also a large proportion of development 
costs / in most expert system development projects, must be 
devoted to knowledge acquisition efforts (Dills, 1989; Brule and 
Bloimt, 1989; McGraw and Harbison-Briggs , 1989). 

The center of focus f'jr the examination of this process in this 
paper is the transfer of conceptual structure from the mind of a 
human expert to the user of an expert system or other 
intelligent, knowledge-based machine. Three areas of scholarship 
within the knowledge-acquisition problem <u:e relevant to our 
center of focus. These will be described, and the way in which 
they relate to the design of expert systems will be pointed out. 
Then an experiment will be described which explores an aspect ot 
these areas using tools which are new to the area of knowledge 
acquisition for expert system building. Finally, implications 
and extensions of the experiment will be discussed, and 
recoxranendations concerning the use of the tools will be made. 



KNOWLEDGE ACQUISITION TAXONOMY 



The three areas of focus are related to each other and to 
the variables Depth of Structure, Type of Knowledge, Methods for 
Elicitation and Cooperative Arrangements in the accompanying 
figures. 

The first area is that of culture. In this area, the 
problem becomes one of eliciting knowledge from the environment, 
both social and non-social. The people working on this problem 
are the anthropologists, a classic example of which is presented 
by H.G.Barnett in his book. Innovation: The Basis of Cultural 
Change (1953). Contemporary examples of the application of the 
findings from this area to the design of expert systems can be 
found in the archeological work of Benfer and Furbee (1989), the 
examination of current theorizing on the family as an institution 
(Sprey, 1988) the Italian project for socio-scientif ic 
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knowledge acquisition (Bianco and Rolandino, 1988). Arandelovic 
(1986) has investigated the role of Utopian thought (versus 
empirical thought) in knowledge acquisition. Within the applied 
technical realm, work such as that by Priest (1986) on the 
difference in cognitive strategies by novice and expert physics 
problem solvers belongs at this level. 

The second area is that of child development and world- 
view/construct formation psychology and the sociology of 
knowledge. Here the problem becomes one of describing how a 
child elicits knowledge from the world aroimd him, or how the 
adult learns from others or from the world, and how they 
structure this elicited knowledge. Currently, fc example, work 
is being conducted using the work of Piaget (Siegel, 1978; Venn 
and Walkderdine, 1978; Blazek, 1979), Thomas Kuhn (Gagnon, 1^78), 
Niels Bohr (Blazek, 1979) and others. Work has occurred in 
predicting whether or not knowledge acquisition will occur in the 
clinical setting, for example, by Sadowsky and Kunzel (1988), and 
both modern and classical theories of knowledge acquisition are 
described by Haroutunian-Gordon as she examines how psychology 
explains behavioral and cognitive change (Haroutunian-Gordon, 
1988). Andersen (1984) has examined the acquisition of socio- 
linguistic knowledge by children. Knowledge acquisition in a 
developmental perspective as seen through the study of life-span 
development has been studied by Hoyer (1980). Knowledge 
acquisition as a social phenomenon, and as an activity driven by 
social interactivity, has been examined by Garley (1986). The 
problem of acquiring knowledge from machines, and in particular 
from computers, has been treated extensively under the heading of 
intelligent tutoring. Knowledge acquisition from an intelligent 
tutor has been treated by Mandl , et .al . (1988) , and the same 
problem has been treated extensively by Wenger (1987) in so far 
as knowledge acquisition by students affects the design of 
intelligent tutors. Dills and Romiszowski (1990) are designing 
an instructional design model for interactive instruction based 
upon what is currently known concerning knowledge elicitation and 
acquisition by students from intelligent tutors and other 
intelligent, interactive sources. Ford, Petry, Adams -Webber and 
Chang (1990) have extended our understanding of the acquisition 
of deep-structured knowledge using Personal Construct theory. In 
a related approach. Dills and Miller (198 ) have used the 
semantic differential as a basis of deep-structure knowledge 
acquisition in the affective domain. Papers by Hoyer (1980), 
Fitzgerald (1980) and Dusek and Meyer (1980), dealing with 
psychological theories of knowledge acquisition within the 
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framework of life-span development research, contain suggestions 
of potentially great value to theory and practice of knowledge 
acquisition and structuring within the third area of our focus, 
described below, which have never been exploited. The same can 
be said for the work of Carley (1986), especially through 
application of her model of knowledge acquisition to both 
cultural and technical Centers of Focus. 

The third area of focus is the technological- Here the 
problem is visualized as how knowledge is acquired by one 
technician from another, or from written documents containing the 
knowledge to be acquired, or from data gathered from monitoring a 
process or technical instruments. This area of focus 
incorporates aspects of the first two areas, but modifies their 
import with the assumption that all parties involved in the 
knowledge acquisition process are aware of their involvement and 
are attempting to be cooperative (at least until something 
happens to negate this cooperation). This is the area :hat is 
usually associated with the term "knowledge engineering", but it 
also includes other areas, such as task analysis and content 
analysis within the instructional design and development 
paradigm, as well as other areas not normally thought of as 
related to knowledge engineering, such as auditing as practiced 
by accountants, legislative research as practiced by congressmen 
through junkets, open hearings and listening to lobbyists, and by 
spying as practiced by CIA intelligence analysts and the 
popularized versions of crime detectives and private eyes. 

Reviews of techniques and heuristics within the third area 
of focus have been providea by a large number of authors, since 
this is a rapidly changing field. Anderson (1988) provides a 
review of the basic techniques employed in expert systems 
construction within the library science and information fields - 
Wiggs and Perez (1988) perform a similar service for the 
instructional development and design, instructional materials 
production, and cognitive science research fields. More in-depth 
coverage of knowledge acquisition within this area of focus, 
especially concentrating upon expert systems applications, are 
found in Wenger (1988), McGraw and Harbison-Briggs (1989), Brule 
and Blount 1989), Hart (1986), Parsaye and Chignell (1988), 
Martin and Oxman (1988), and Carrico, Girard and Jones (1989) and 
Harmon, Maus and Morrissey (1988). Case studies within this 
context are presented by many writers, including Brule and Blount 
(1989) ,Benfer and Furbee (1989) and De Jong (1988). 

Work in all three areas of focus have in common that a clear 
distinction between the practice of knowledge acquisition and the 
practices of knowledge creation and knowledge transmission cannot 
be maintained. In a sense, these areas are merely different 
perspectives on the same problem. This can be seen within the 
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first level of focus in the work of Barnett (1953), within the 
second level through the papers of Fitzgerald (1980) and Dusek 
and Meyer (1980), and on the third level through the writings of 
Hart (1986) and Ford (1989) on knowledge acquisition and those or 
Hutchinson (1984), on task analysis and instructional development 
models . 

Work in knowledge acquisition can be classified according to 
other dimensions than that of cultural-to-technological- For 
example, knowledge acquisition efforts can be classified 
according to the depth the desired knowledge lies within the 
expert's (or other source's) cognitive structure. Knowledge may 
be classified as shallow knowledge, conceptual knowledge, deep- 
structured knowledge and meta-knowledge . Another, similar 
classification is given by Brule and Blount (1989), and is 
derived from Bateson (1972). This classification involves 
Learning Levels 0 through III. Level 0 involves responding in a 
fixed manner; Level I involves responding with varying items from 
a fixed set; Level II involves responding with various items fcrom 
a set of generalized rules or principles; and Level III involves 
treating the rules or principles as metaphors. 

The classification system for depth of knowledge to be used 
in this paper involves five categories. One is factual and 
procedural .nowledge. The second is conceptual and principle. 
The third is structured and systematized knowledge, as with 
algorithmic knowledge. The fourth is world-view knowledge, or 
hermeneutic knowledge. The fifth level is neurol ogicall y or 
psychologically based or derived knowledge peculiar to the 
individual possessing it. A point needs to be made concerning 
different aspects of knowledge structures. The depth of 
knowledge on a particular topic is not uniform. This is obvious 
in a sense; terminology is at a more shallow level than is 
understanding of the interactions among process variables. But 
it is not so obvious, but is nevertheless true, that the 
cognitive knowledge of a topic and the evaluative structure for 
that same topic are often at different depths, and are somewhat 
independent of each other (Carley, 1986). Social 1 ycompact 
groups of experts tend to develop identical cognitive structures 
concerning a given topic at a shallow or medium level, but have 
differing evaluations of that topic. Thus it may be useful, or 

even necessary, to categorize not only topics within knowledge 
structures by depth, but also to independently classify various 
aspects of each topic by depth. This will allow the application 
of the appropriate tool or technique not only to each topic, but 
also to each aspect of each topic. 
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Other categorizations of knowledge acquisition work are 
useful- For example, knowledge acquisition may be classified 
according to the social roles involved- The dimension describing 
such classifications will be called Cooperativ3 Arrangements in 
this paper. One such classification would be the followirxg: 
Elicitor cooperating with expert sharing a goal; elicitor with 
expert, not sharing a goal; elicitor with documents; elicitor 
with finitely-bounded real-world situation; elicitor with 
unbounded real -world situation; elicitor with unbounded 
metaphorical situation- Carley (1986) has found that social roies 
and other social variables strongly influence the manner in which 
knowledge acquisition occurs, as well as the manner in which the 
knowledge is structured once obtained - 

Carley has developed a knowledge acquisition model based largely 
upon social interaction variables- A great deal more could be 
done along these lines, especially in some of the more exotic 
cells. A knowledge acquisition management tasks could be better 
structured than they currently tend to be by considering 
knowledge acquisition as a social function rather than as a 
technical writing function- 
One exotic type of knowledge acquisition not included in any 
of the cells of our classification system, and which is best 
thought of in terms of the Cooperative Arrangements dimension, is 
knowledge acquisition from the self- Such a type of knowledge 
acquisition is clearly very closely related to internal knowledge 
creation- And a great deal of research has been devoted to wnat 
might be called internal knowledge linkage, or internal diffusion 
of knowledge- These functions have been studied from many 
perspectives over the years, such as the psychology of creative 
thought, memory retrieval, transfer of learning, and serendipity. 
However, these topics have not been widely applied to the 
knowledge acquisition problem- Obvious areas of application 
involve thought problems for the expert to use in ''he "difficult 
problem" approach to expert systems building, to \trospection 
when an expert is attempting to find why he made a certain 
decision, and to the knowledge engineer when planning scenarios 
to present to the expert- It certainly also applies in the 
situation in which the knowledge engineer and the knowledge 
expert is the same individual- Several chapters of most 
contemporary textbooks on cognitive science or cognitive 
psychology are typically devoted to these issues- For example. 
Cognitive Science: An Introduction (Stil lings, Peinstein, 
Garfield, Rissland, Rosenbaum, Weisler and Baker-Ward, 1989) goes 
far as to devote a complete chapter to problems of knowledge 
structure and self -learning machines, and several other chaptets 
touch upon these questions- However, the tradition in psycholoyy 
of studying introspective knowledge acquisition goes back at 
least to the introspective psychologists of the 1920s and 1930s. 
The problem of machine generation of rules from internalized 



s 



DILLS MfD ROMlISZaWSKI 



performance data is being treated through inductive generation ot 
theorems and heuristic rules in contemporary computer science 
(Carrico, Girard and Jones, 1989)/ and was treated by Dills and 
Popplewell within the context of intelligent inductive rule 
generators in military maintenance training simulators during the 
early 1980s (Dills and Popplewell, 1987). 

A final dimension of importance in classifying knowledge 
acquisition efforts is by tools used by the knowledge elicitot . 
The most fundamental level is that at which only organizational 
and recording tools are used. Interview techniques, tape 
recorders, video recorders and flowcharting are at this level. 
The next level is that at which software tools are available. 
Most of these tools are very much like the previous set of tools, 
except, from the knowledge elicitor's poiL of view they are nor 
manually operated (although they may be manually operated trom 
the point of view of the expert from whom they are eliciting 
knowledge). The simplest of these tools merely let the expert 
enter knowledge directly into the knowledge base of the compute=^r. 
The most complicated are the inductive tools, which generate 
rules from examples input by the expert (Carrico, Girard and 
Jones, 1988). Unfortunately, xione of these tools can adequately 
deal with metaphors, world views, diagrams, pho^:ographs , 
multiple-expert disagreements, hunches, and so on. Therefore, 
there are many knowledge acquisition problems for which no 
adequate software tools exist. Therefore, the next level ot tool 
use involves the use of language analysis, philosophical 
analysis, psychological testing, attempts to transmit intuitions, 
and just plain thought as tools. These tools are often 
inadequate, of course, but at the present time they are all that 
exist • 

Thus, we can describe any knowledge acquisition study or 
project according to the cell it occupies within a rour- 
dimensional matrix defined by the dimensions Area of Focus, Depth 
of Knowledge Sought, Social Roles and Tools available. Certain 
of these cells are well explored, and problems occupying them are 
quite amenable to commercial solutions. Others are on the 
borderlines of commercial feasibility, and books on the 
construction of expert systems expend efforts to show the project 
manager how to identify projects from these cells. They then 
explain why they should be avoided, in terms of the current state 
of the art (Martin and Oxman,1988; Carrico, Girard and Jones, 
1989; Harmon, Maus and Morrissey , 1988 ) . Applications trom sti i i 
other cells at the moment are completely absent from the 
commercial world, either because there are no commercially 
feasible ways to deal with them, or because there is no 
coiwnercial market for the type of knowledge acquired within these 
particular cells (or both). 
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The authors believe that eventually the state of the art ot 
expert system utilization will develop to the point that 
knowledge acquired in all of the cells will be needed. When this 
occurs, the state of the art of expert system design, and of 
knowledge acquisition, will be required to advance to the point 
of being able to work with problems from all of the cells. 
Furthermore, any methods we develop capable of working in those 
cells which at the moment present the most difficult tasks wxli 
surely have significance for commercial projects • Therefore we 
are searching for knowledge acquisition methods to apply to deep- 
structured, world-view types of knowledge, which lie within the 
first area of focus. 

One tool we believe to be likely to be useful at this level 
is derived from structural conmiunication . 



STRUCTURA L COMMUNICATION 



Structural coiratiunication is a particular model of teaching. 
As such, it is a self-contained system, complete with information 
delivery system, remediation units, evaluation system and 
feedback mechanisms- Its uniqueness stems from two goals of the 
system: it strives to be both sel f -instructional and 
automatically administered, and it strives to engage the student 
in what most taxonomies of learning would term the "higher" 
stages of learning. Originating early in the 1960s as an attempt 
to merge certain of the theories of J.S.Bruner and B.F. Skinner, 
among others, the result has become a very flexible but 
structured model for teaching through guided dialogue 
(Romiszowski,1896; Eagen, 1976). 

The basic unit of structural communication is the study 
xmit. A study unit typically consists of six interdependent 
parts, although it is not necessarily inherent to the nature ot 
structural communication that all of these parts be present in 
every single study unit. The six parts are as follows 
(Romiszowski 1986) : 



1. Intention. This telJs the purpose of the 
study unit, presents its objectives, and 
serves as an advance organizer. 

9 
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Presentation. This section introduces the 
student to the contents of the unit. it 
serves as the lecture , text assignment , or 
other information presentation element oi. tne 
study unit. 

Investigation. This section usually consists 
of some problems concerning the topic of the 
study unit that involve the information or 
concepts presented in the Presentation 
section. The student attempts to solve or 
otherwise deal with these problems, and 
presents this solution in the next sect i on . 

Response Matrix. The response matrix is a 
randomized array of between 10 and 35 
elements. Each element represent r> a 
significant aspect of the topic of the study 
unit, an element from the presentation, a 
part of the solution to the problems from the 
Investigation section, or a distraotur. 

The student must select the items that 
represent his answer to the question oi 
solution to the problem posed in the previous 
section. The student does this by seloctincj 
the items in a particular sequence . For 
example, the task might represent the 
sequence of steps in solving a problem, or 
the topic sentences from an essay (in the 
order they would appear in the essay). 

Discussion. The student is then provided 
feedback, comments or further dialogue with 
the instructor in the Discussion section. 
The actual mechanics of how this is 
accompl ished varies considerabl y t rem 
application to application, and is largely a 
function o^ the medium being used (i.e., 
paper and pencil, computer, Hash cards, 
audio tapes, etc.). In all cases, the 
general idea is the same, however. The 
comments the student will rece}ve have 
already been prepared, and have been grouped 
according to the types of choices the student 
conceivably could make. 
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Suppose the student selected items 7, 
8, 9, 2, 22, 18, in that order. Suppose t hr- 
instructor agrees that items 3, 6 and 7 are 
important and should be tiist in the sequov.'Mr 
of choices. The student will receive a 
message or messages telling him this, and 
possibly stating the reasons, II the 
instructor thinks that item 8 is a 
distractor, the student will receive a 
message telling him the instructor believes 
8 should have been omitted, and telling wny. 
An invitation to the student to disagree, or 
to discuss the matter in person or by e-mail, 
likely will be included. If the instructor 
agrees that items 9 and 22 both should have 
been selected, but in reverse order trom i r- i 
used by the student, the message will explain 
why this is so. Finally, suppose the 
instructor agrees that item IB should be 
selected, and in its current position, bin. 
believes it has meaning only it selected 
along with item 23, which the stuuenL 
omitted. The message will explain the 
relationship between the two items, ana Atgu*-j 
for their joint selection, 

6, Viewpoints, This section presents i he 
instructor to the student. Here the 
instructor points out what aspects ot the 
dialogue are subjective on his part; what 
parts are controversial and describes 
alternative viewpoints to those taken in the 
lesson. The student is also guided to 
further exploration of the study topic in 
this section. 



It should be clear that often the choices evaluated in the 
student's response are value judgements or subjective decisions. 
This is not always so, but structural communication lends itselt 
quite readily to the study of subjective questions, and so is 
often used in that context. In any case, the responses given to 
the student are tailored to his original response, and take the 
form of a dialogue between the student and the instructor 
concerning the student's response, just as at a higher level ol 
structure, the entire study unit is a dialogue between the 
student and the instructor about the subject area under 
examination. The dialogue can be extended through several 
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sequenced or related study units / forming a course or curriculum. 
Or it can serve as an isolated experience, either exploring a 
topic or serving as a final integrating experience following a 
more traditional introduction to a course. 

While the six components of the study unit described above 
were the parts of the study guide as originally conceived, ana 
still today can be fo\md in almost all applications, they are not 
essential to the concept of structural communication. What is 
essential to this concept can be found by examining the function 
these parts play in the study unit. Finding this essence is 
important to the purposes of this paper because the use of 
structural communication as a knowledge acquisition tool r equi r f-s 
modification of the above six~part format without violating the 
basic concepts involved, and using these concepts for a purpose 
other than teaching. 

Structural communication, in any of its forms, may be 
defined as "the staged presentation of complex instrucrionai 
materials to a student, together with devices to encourage his 
engagement in these materials at the higher stages of 
intellectual functioning, followed by an assessment ot the 
intellectual structures in the student that have resulted from 
this engagement, further followed by corrective and suppiemeatai 
feedback from the faculty, again presented at the same high level 
of intellectual functioning." 

Not necessarily required by the concept of structural 
communication, but typically a practical necessity in order to 
implement the full set of goals and requirements of the concept, 
is the operation of the system in a self-instructional, seit- 
paced mode, and is often delivered via a computer. 
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THE EXPERIMENT 



An experimeDc was undertaken to determine whether or not the 
deep-knowledge structure of a subject-matter expert could be 
discovered through the use of the structural -communicatiop 
knowledge-acquisition approach- Three stimulus sheets were 
prepared, each proposing to teach the topic, ''American Popular 
Culture", to the junior and senior high school students of rural 
traditional areas of Formosa and Korea. Curriculum content was 
not described, but the use of high technology delivery systems, 
independent and self -directed study methods, criteri on-baseii 
testing and other such uniquely-American teaching and learning 
strategies were specified in detail- One stimulus sheet was 
centered upon CBI, a library and interactive video discs; one 
used distance education via satellite; and one involved 
interacting with an expert system, a library and video discs. 

Four experts were chosen from the students enrolled in a 
Syracuse University graduate class whose purpose was to explore 
the likely impact upon education of various innovative higi^ 
technology communications systems, IDE 716- Three of these 
experts were students from Formosa, one was a student trom Korea. 
These students were pursuing a master's degree in educational 
technology except one, who was pursuing a similar degree in the 
department of speech pathology- 

The remaining students in IDE 716 and their instructor acteu 
as knowledge engineers (two American students and a British 
professor)- Their ultimate task was to discover the structure 
and content of national belief system concerning education ot the 
Taiwanese and Korean peoples- Their immediate task, which » l- 
assumed to be the first step towards accomplishing this tasic, wa.s 
to discover the structure and content of the belief system heiu 
by each of the subject-matter experts within the class. They 
used the structured-communications approach to do this. 

Each knowledge engineer interviewed each of the tour 
subject-matter experts- All knowledge engineers used an 
identical 57-item selection matrix and annotation sheet, but each 
knowledge engineer used a different proposal description. 

Each interview was based upon the same scenario- The 
scenario claimed that the expert's nation had issued an RFP to 
companies in the United States, asking them to submit a propose i 
for a course explaining American popular culture to the public^ 
school students of their country- The company is to describe the 
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method for teaching the course well as content, and high- 
technology methods are to be used whenever feasible. The project 
will be financed by an American government organization, and wi i i 
serve as a model of technology utilization, so cost linatatinns 
are not to be considered a factor in deciding among propoi:>aJG. 
Each knowledge engineer represents a different company, and the 
proposal descriptions are the technological components ot each 
company's response to the RFP. Each knowledge engineer asks the 
expert's help in determining whether or not anything contained in 
the proposal conflicts with any practice or belief concerning 
educational practices within the rural, traditional view or 
education held by the students, teachers or parents or the 
regions targeted by the proposal. 

Subject-matter experts were instructed to read the proposal 
before and again during the interview with any given knowledge 
engineer. Then, with the proposal before them, they were to i o 
each of the 57 items on the selection matrix and decide ii: thv 
item applied to the current proposal and also if the itea? 
violated some belief, custom or practice of the ^ uituie tne 
expert represented. If so, the expert was to draw a line thm^^K 
the item on the selection matrix, and to describe the reason ror 
rejecting the item in a sentence or short paragraph on the 
annotation sheet. Thus, in this particular version of sttuctured 
communication, the expert was given what in traditional writmor^ 
on structured coiranuni cat ions is called the case study and the 
selection matrix, and was asked to make selections on the 
selection matrix and write the feedback discussion themselves. 

Following the completion of the selection matrix and 
feedback comments, the knowledge engineer would read it over ana 
conduct a de-briefing interview with the expert. During chis 
interview, which was open-ended and free form, questicms about 
items not eliminated on the selection matrix, as well as about 
items eliminated, were to be asked. No record of this interview 
was kept except to the extent that it resulted m the subject- 
matter expert modifying the written response record (when 
instructions had not been understood, or a question arose as i o 
the meaning of an English phrase on the selection matrix, tor 
example). The interviews were discussed in class, however. 

The selection matrices and annotation sheets were signed by 
both the knowledge engineer and the subject-matter expert 
following each interview, and were turned over to the author rur 
processing . 
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Processing occurred in several sequential steps. The tirst 
was to tabulate the items on the selection matrix that Wf-rc 
eliminatGd by each of the experts, and to determine the degree to 
which they agreed. The degree among experts across knowledge 
engineers, and of all experts within a given country, were also 
determined (thereby also determining the degree of disagreemeni. . 

Next, item reduction was performed- This process consisted 
of combining items for which the explanations entered on th*^ 
annotation sheets were identical, or very close to being the 
same. The explanations were then re-stated in such a way as ici 
attempt to elucidate the principle or structural component ot the 
\inderlying knowledge structure. This was a qualitative, not a 
quantitative process, and is subjective. Nevertheless, the data 
in the pilot study described here seems clear-cut enough to 
justify this procedure. 



OUTCOMES 



SUBJECT-MATTER EXPERT 



TOTAL 



K 
W 
O 
W 
L 
E 



E 



A 


11 


8 


16 


5 


1 

40 I 


B 


8 


18 


17 


10 




C 


7 


15 


25 


12 


59 


TOTALS 


26 


41 


58 


27 
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NUMBER OF RESPONSES : The niimber of responses obtained by 
each of the three knowledge engineers differed. This was true 
not only between subject-matter experts from different countries, 
by from experts within a single country (Formosa). For example, 
one engineer received 8, 16 and 5 responses from the three 
experts from Formosa. A second engineer received IH r 17 / and io 
responses^ and the third engineer received 15, 25 ana 12 
responses. While it is inappropriate to compare responses or 
subject-matter experts across engineers quantitati vely ^ since 
they were responding to different problems with each ot the 
engineers, a qualitati^'^e comparison shows that some subject- 
matter experts have a tendency to respond more frequently than ao 
others. This fact indicates a need to use multiple experts, and 
also a need to follow up the procedure with de-briefing 
interviews and a further round of structured communication 
problems in order to probe the deep-structure knowledge to a 
greater depth. It also probably indicates a need for more 
probing when using other methods of knowledge acquisition, since 
this tendency on the part of subject-matter experts is probabiy 
not limited to the structured-communication environment. 



CONSISTENCY ACROSS EXPERTS: Each subject-matter expert was 
faced with the identical problem from a given knowledge engineer. 
If we assume that each expert is identically informed concerning 
the culture and educational institxitions of his or her country, 
we would expect that all experts would respond identically on a 
given problem. The variation among knowledge engineers shuiuii 
not matter, since for a given problem, only a single knowledge 
engineer is involved. However, this did not happen. Tne degree 
to which consistency of response failed to occur can be taken as 
a measure of the error of measurement of the procedure, and ivs 
inverse as the measure of inter-subject reliability ot the 
procedure. One knowledge engineer obtained 31 responses tioxu 
three subjects from Formosa. Of these, 12 were given by two or 
more experts, and none were given by all three. This represents 
a 38.7% duplication. Thus the inter-subject reliability tor that 
problem can be taken as .387. For another engineer, with o 
different problem, over the same experts, 33 responses were 
obtained, with 11 being given by two or more experts, an'i l\yr. 
being given by all three experts. This produces an inter -sub jecL 
reliability of .333, The third engineer received 21 refnonses, 
with 8 responses being given by two or more experts and none 
given by all three experts. Again, this was a different ptobiem 
from that used by the other two engineers. This resulted in an 
inter-subject reliability of .381. The average of these 
reliabilities is .367. The figure given when the data is from 
each engineer is combined is .381. Thus, the inter-subject 
reliability for the method can be considered to be approximately 
around one third. 
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A follow-up experiment is suggested by this data. A second 
round of interviews could be conducted by the same knowledge 
engineers and the same experts, asking each expert to comment 
upon the responses of the other experts. Thus it would be 
determined whether or not the low reliability value is a result 
of disagreement among experts or is a result of individual 
differences in responding. If it is a result ot individual 
differences in responding, then such second-round interviews 
should be routinely used in order to obtain consistent responses 
and to obtain them from a consistent depth in the experts' n^r 
structures. However, if the differences in responding is a 
result of differences in the mental structure ot the experts, 
then some other method needs to be used to determine the aclucii 
situation. The engineers would be on their way toward mapping 
the cognitive structure of the experts using the method under 
investigation, but this would not necessarily lead to an aocuiai^ 
and tisable map of the reality the mental structures mirror- 
Unfortunately, this follow-up study could not be pertormed 
because the experts and some of the knowledge engineers are no 
longer available for conducting the study. But this approach 
should be followed in future experiments. 



CONSISTENCY ACROSS KNOWLEDGE ENGINEERS : No attempt was mad 
to determine the degree of reliability of results across 
knowledge engineers, since each knowledge engineer used a 
different problem than did any other. In the future, some 
attempt should be made to determine the degree ot this 
consistency- This would be determined by an experiment simiiat 
to the one described in the current paper, but with ail knov?iedc 
engineers using the same set of problems with the same experts. 



COMPOSITE PATTERN NOTES : Composite Pattern Note^ were 
produced in a variety of ways. Composites were produced tor eac: 
subject-matter expert across engineers, and for each (-i^nint.'-- 
across experts for a given country. Finally, Composites were 
produced for each country using all experts from that country. 
These, again, were not verified to\ expert a«ji ^ \ using r 
Delphi or some similar method. It is recommended » I . » «n « - 
future this be done. 

The cotuitry Composites were compared to determine it the two 
countries were r<ipresented by distinctly ditleteni mental 
structures in tht minds of the experts. All experts combined 
responded to 47 Ov't of 57 items. Experts from Formosa rei^pi^vio?-) 
to 43 items, and the expert from Korea responded to ib itemr>. 
Three of the items responded to by the Korean expert, were noi 
responded to by the Formosans, and 28 of the items responded to 
by the Formosans were not responded to by the Korean. inererore 
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there were 31 items out of 47 that serve to di f terentiate tho zvr^ 
countries. This represents 66% of the items responded to. irii: 
means that two-thirds of the items receiving a response are 
discriminatory items. However, it is not true that two- tniras (^t 
the responses were discriminatory. There were a total oi: 151 
responses by experts. Of these, 82 were to items that were iatr; 
discovered to be discriminatory. This represents 54.^% oi the 
total number of responses. So while two-thirds ot the items wer^- 
discriminatory , only about half of the responses were 
discriminatory . 

A Delphi or some similar method should be used to verity that 
differences in the Composite Pattern Notes for the two countries 
truly reflect conceptual differences in the minds of the experts 
are a result of individual differences in response generation, m 
in the depth from which the responses were extracted trom the 
different experts • 

QUESTION generation : Assuming it to be tound thai- 
differences among the Composite Pattern Notes tot the i.vru 
countries represent real differences in the mental structures w. 
the experts, the next round of knowledge acquisition can o-rj 
This effort would explore further the results already ototameu. 
The direction of this exploration would be determined tn?:^ 
motivating needs of the project tor which the knowledge i:j ije^iy 
acquired, and could either be directed toward v.hf> prodiicl. i rn? 
more detailed knowledge or the verification ot general i zationr; . 
In either case, response items can be generated trom M^e 
Composite Country Pattern Notes, and these in turn can be us-u lo 
outline the scenarios. 



FUTURE RESEARCH 



Several studies need to be carried out iu Iht- nature in 
order to determine the exact circumstances in which various 
versions of structured communications are appropriate tor 
knowledge acquisition. The first such experiment, ot course!, js 
to extend the present study to the second stage, in wuioh 
questions are generated from the present data and presenteo to 
the same subjects to try to map the cognitive structuro tn a 
deeper level. A second, and related, experiment is tc present 
the same stimulus material used in the present expot iwvjri i '^ /w 
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subject-matter experts in order to determine the cunsii:^ te:u y 
their views with those of the present experts. If sigmli.cani 
differences are found, the use of structured conffnunications ir, ^\ 
Delphi-type procedure could be tested as a way uu recoiK^i e 
differences among experts. 

Finally, a series of studies are contemplated in which t ht: 
structured communications form.s that are tound to bo tht^ tno;-. 
useful in knowledge acquisition tasks would be combined with 
various versions of computer-mediated cotnmunications systems, 
such as E-Mail or networked files on IBM mainframe CMS systemii. 
Attempts would then be made to use structured coim>.unicat.i on s i - 
reconcile or acquire knowledge in a number of situations or 
potential value. For instance, it could be used to acquire 
knowledge on a topic from a number of experts in different 
locations. It could be used to acquire a consensuti ot expei r 
opinion on a topic from the entire expert community of the UniUi-d 
States. It could be used to reconcile expert opinions ami^ug 
geographically separated experts who differ in their viewpointr. 
And it could be used in a particular community (t^uch as a 
classroom, a school, a company or a research group) to i:>uiifi 
an evolutionary fashion a knowledge structure reprepeni iili v*- v 
the entire group. 

The questions to be asked in these experiments are: 



(1) whether the application works. 

(2) under what conditions does it work bettor 
(faster, cheaper, more eff i ci f-r<. 1 v r^^i i n 
greater depth, attain easier reconci t la i on , 
etc.) than do alternative knowle<icie 
acquisition methods? 

(3) can unique knowledge acquisition goals be 
achieved that cannot be achieved any otherV 
and, (4) are there better analytical melhou: 
available for use in analyzing the dat?:2 and 
formulating the resulting knowledg-^e si r act uti- 
than pattern noting, especially within trie 
context of computer-mediated communica r 



Another area that needs attention is the area ot the 
knowledge-acquisition process from the point of view of the 
expert whose mental structures are being studied- The importaiice 
of this is shown by representing the knowledge-acqm 1:3 on 
process in terms of the Shannon-Weaver communications moaei 
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(Shannon and Weaver, 1964). In this model, the process consxia:' 
of a transmitter and a receiver, with a transmission mediur^ 
between, and with static or other noise inherent to the 
transmission line. The sub ject-ma ter expert is the tr anr^^nti - j . 
and the knowledge acquisition expert is the receiver. Buv how 
does the expert decide what to transmit? How ciuejs h^s ht-v.' : v 
structure and other mental structures aftect this decisicu: hiu.i 
how does he view the task he is engaged in? There has been * 
wide range of answers to these questions exhibited within 

contexts in which I have worked. Sometimes the expert ii^ 

against his will, sometimes the expert has originated the 
project. Sometimes the expert wishes to hold back c^^\.ni: 
information, or all information beyond a certain depth ot 
expertise. Sometimes the expert tells the knowledge acqm 
expert these things, sometimes these things are held hacK am: 
must be guessed. 

The Johardi Window, used to model inter-personal 
communications (Coscarelli and Stonewater, 1984; Lutt, t^bx) 
might be a useful tool for picturing the questions raised by 
introducing the expert's personality as a tactot in vfh.M 
knowledge is accessed. In the johardi window model , eacti 
participant in the communications process is speaking ihrou'j;. » 
window. This window is partially open, and has a shad«^3. Behind 
the window is the person's mind. Part ot that mind is dirpol y 
transmitted through the open window. Part is transmitted, bur. ii 
a distorted form, through the glass of the closed part or th^^ 
window. And part is never transmitted, because ot uhe oi'um. 
The same happens to information arriving at the window tio:n 
outside. Part is admitted, part is admitted in a aistortr-^i w.imij 
and part is blocked. Further, the window and the blloM jr*- 
movable, partially at the control of the person behind thr 
window, and partially by other factors. Thus when tne- 
talking to a subordinate the window is open a certain amcunv , v 
when talking to his superior, it is open a ditterent amou: ' . 
does this window function between an expert and a knowxeoge 
acquisition specialist? What factors of management, ol 
personality, of procedure, and of acquired ownership of i h-. 
project, effect the window? How do they effect it ' 

The phenomenology of the expert needs to be examinea, 
especially in relationship to knowledge acquisition proje^^^ i: y-- 
expert system production. For example, how does an expert L'-mq 
to understand the knowledge acquisition process? How dues h! 
expert tmderstand the nature of his own expertiseV As tat a.o i 
have been able to discover, these questions have nevor oe^.i: 
investigated. One way in which do conduct such an investiuat ^ on 
would be through the use of conventional intervievj ttji-jhniquer . n 
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to ask an expert to write a essay discussing his view ot the 
matter. But if more than superficial knowledge or Lhe export <: 
feeling and viewpoint are to be acquired, a more subtle rina 
sophisticated approach will be needed. 

The structural communication approach couid be used as a 
research methodology for investigating these questions reiotuxc 
to the expert's view of himself, his expertise and the 
acquisition process. The procedures to carcy "^it t,hci)v " 
would be much like the procedures used in the experjr»v-ui 
described earlier* 
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DEPTH OF KNOWLEDGE 



1. FACTUAL AND PROCEDURAL 

2. CONCEPTUAL AND PRINCIPLE 

3. STRUCTURED AND SYSTEMATIZED 

4. WORLD-VIEW (HERMENEUTIC) 

5. NEUROLOGICAL/PSYCHOLOGICAL 
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VARIABLES (DIMENSIONS) OF 
KNOWLEDGE ACQUISITION 

1. DEPTH OF STRUCTURE 

2. TYPE OF KNOWLEDGE 

3. METHODS FOR EUCITATION 
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SOCIAL ROLES 

1. ELICITOR WITH EXPERT, 
SHARING A GOAL 
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KINDS OF KNOWLEDGE 
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